Until recently, those in the tele-robotics community, namely the roboticist and control theorist, have dealt primarily with the sensing of the physical environment, measuring positions, forces and accelerations, and responding with movements and forces to directly manipulate the physical or work environment by use of a mechanical operator or manipulator such as robot. A human operator provides the cognitive power of a tele-robotic system and directs the movement and function of the manipulator or robot. Such systems are described in detail, for example in J. Vertout & P. Coiffet, "Tele-Operations and Robotics: Applications and Technology," Robot Technology, Vol. 3B, English translation (Prentice Hall, 1986). The term "telerobotics" or "tele-operation" is used particularly where the human operator is in a master environment which is separated from the physical or work environment where the manipulator or robot is, and telecommunications, often including a video link, are provided so that the human operator can monitor properly and control tasks being performed in the work environment. Often, these two environments are separated by shielding, as in the case of remote manipulators being used in hazardous waste sites, nuclear power facilities and the like, or by a hostile environment such as the deep sea in underseas exploration or the vacuum of outer space. One hallmark of such tele-operation or tele-robotic systems is that the sensory-motor processing of the human is intermediated and projected at a distance by the controlled machine. When tele-roboticists discuss the projection of "autonomous intelligence" using such remote mechanisms, the projected capabilities are usually envisioned as programs that can be invoked to carry out, independently of active human intervention, physical manipulation tasks, with the human remaining in contact and control only at a supervisory level so as to be able to interrupt or intervene in such tasks when something has gone awry. See T. B. Sheridan, "Human Supervisory Control of Robot Systems," Proceedings of the IEEE International Robotics Conference, pp. 808-812 (April, 1986). On the other hand, another common goal for tele-robotics research is to provide slave robot system where the human is the only cognitive agent actively directing and controlling all aspects of the work in the remote environment. In such systems, the goal is the production of as realistic a sense of remote tele-presence and tele-control for the human operator as possible, given physical constraints such as communication delay times. The ultimate goal in such systems is to enable operators to do as nearly as well at manipulation tasks as they could do if physically present at the remote location.
In the past few years, there has been a rapidly growing interest in autonomous systems research. Those active in this area are computer scientists and artificial intelligence researchers who typically are working to produce self-contained, mobile platforms, such as the autonomous land vehicle (ALV) and various autonomous underseas vehicles, that can maneuver around and employ machine cognition to seek high-level goals in their environments. The focus here is on mechanization of sufficient machine cognitive power to achieve goals, such as complex route planning and replanning, to effect reconaissance or other information-gathering missions, and on providing sufficient perception and maneuvering capability to do things like follow roads, avoid obstacles and find specific objects in the environment.
Given present limitations and computational complexity of self-contained machine perception such as machine vision, the sensory-motor aspects of autonomous systems technology are currently rather crude when compared to tele-robotics, which can exploit human perception. Thus, the autonomous systems community tends to focus on widening the exploitation of machine cognition on tasks that are feasible within the envelope of available perception technology, but only through the use of cognitive interaction with the environment at a symbolic level. A common goal of autonomous systems research is the mechanization of cognition and the associated task-dependent knowledge systems so that the remote machine is as smart, robust, knowledgeable and persistent as a human might be in attempting to carry out its mission. However, since the focus of this work is on autonomy, human supervision or interaction is seldom stressed. This is due, in large part, to the tremendous complexity of providing such interactive capability. Similar levels of complexity are encountered when attempting to enable a much more powerful cognitive agent possessing artificial intelligence (AI) to interact in real time at the problem-solving level with an automated work robot or other manipulator. Thus, when the notion of supervisory control appears in autonomous systems, it usually is concerned with having the human or AI cognitive agent intervene if the system is "not smart enough" to cognitively handle a given situation. Consider for example an ALV driving down a remote road which suddenly encounters uncertain footing or gets stuck, and does not have sufficient exploratory behaviors and learning capabilities to get itself out of trouble, i.e. get unstuck. A skilled "tele-driver" is required to free it. Heretofore there has been no methods or man-machine systems which enable a human to easily "slip into the cockpit" and take over in mid-maneuver to correct the troublesome situation or avoid it altogether. Intervention into such an on-going autonomous manipulation task will normally not be easy, since taking over in mid-maneuver typically involves smoothly effecting a multi-dimensional control rendezvous. There are other examples of the need to provide human supervisory control to correct problems beyond the capability of the automated equipment, such as are being uncovered by NASA scientists as they seek reliable ways to automate future space station activities.
At a more earthly level, the use of teaching and playback modes for programming and operating industrial robots is well known. Vision systems have also been used in conjunction with robots for such tasks as picking up parts off of a conveyor line, sorting, and visual inspection for defects. Programming robots with joysticks during a teaching mode for subsequent repetition at higher speeds in real time is also well known. However, to the best of our knowledge, none of these types of applications involve on-line, real time path planning and active supervision of its operation by a cognitive agent such as a human being or similarly intelligent AI system which can intervene into an on-going manuever and in real time assume control of the robot's actions in the work environment, especially in order to handle unstructured tasks or to retain the flexibility to perform a task much differently than the way the robot had been previously programmed to perform it.
The following patents assigned to the Nordson Corporation provide an example of the type of work which has been done with multiple degrees-of-freedom industrial robots that can be programmed to execute a series of arbitrary motions within the physical constraints imposed by the links, actuators and power sources of the robot:
______________________________________ U.S. Pat. No. Inventor ______________________________________ 4,360,886 Kostas et al. 4,481,591 Spong 4,484,120 Olex et al. 4,484,294 Noss 4,486,843 Spong et al. ______________________________________
The first three patents disclose a robot controller for a work robot which includes a specially-programmed micorprocessor and random access memories (RAMs) for storing program sequences of desired or command positions for the various robot links, as well as the sequences of actual positions of the robot links which result when the work robot is driven by the stored program sequences stored in the RAM. The use of joysticks to program the robot during the learn mode and of two operators working together to program the robot in a training session is also disclosed. The fourth patent discloses a form of real time editing of the stored program as it is executed by the robot. The editing is accomplished by modifying the individual position commands associated with a given robot link through incrementing or decrementing means responsive to the actuation of a manual "positive increment" switch or "negative increment" switch. The last patent discloses the concept of adding sequences of position commands to the end of an original sequence of stored commands. The patent also discloses techniques for connecting up the end of an original sequence of command positions with the beginning of a subsequent sequence of command positions by interpolating to achieve a substantially uniform rate of robot motion. When this rate is the maximum possible rate of the robot joint, the resulting transition between the sequences is said to be traversed in a minimum amount of time by the robot. The foregoing patents are indicative of the complexity of robot systems, and the difficulties presented when attempting to modify existing robot systems to have real time programming capability. The foregoing patents are also indicative of the perceived but unsolved need for such real time supervisory capability by a human being or AI cognitive agent embedded in the control system so as to be able to actively assume control from the robot at a detailed level very close to the work environment.
In some tele-robotic situations, such as an automated space station or space satellite with controllable manipulators operated from an earth-based control station by telemetry, one important constraint on the operation of tele-autonomous systems is the time delay for communications between the local (earth) and remote (space) system. M. Noyes and T. Sheridan of the Massachusetts Institute of Technology have devised a novel way to cope with such a delay by using a locally-situated forward-in-time tele-robot simulator, and a grahical "predictor display" overlay of the forward simulation onto the fixed-delay return video of remote tele-operation. The Noyes and Sheridan system and method is illustrated in FIGS. 1 through 3, which will be discussed as part of the introduction of the present invention in the Detailed Description below. The experiments of Noyes and Sheridan with their system have shown that the time to perform manipulation tasks in the presence of significant communication delays can be reduced by exploiting such predictor displays, as graphically shown in FIG. 2. However, the operator of such a system is still constrained by the time-synchronized controls, i.e., the remotely located robot can move no faster than the operator can safely direct it to move. One example of this is where an operator must go slowly to properly position the end effector on the robot arm carefully about a small object by the human operator, the performance of the system is still very slow and limited, not by the robot, but by the human operator. It occurred to us that it would be highly desirable, in terms of improving the overall efficiency of such a real time system, to allow the operator to enter, while on-line with the robot, a sequence of commands faster than the tele-robot could carry them out, and then allow the robot to perform the recorded sequence. In other words, we hypothesized that in a real time system it would be very desirable to provide kinematic and/or dynamic forward simulation of at least portions of the work environment which are not constrained by time synchrony requirements. Also, we felt it would be very desirable to free the operator from close attention to the robot's movements when such attention was in fact not needed.
As the foregoing discussion indicates, there is a long-felt but unsolved need for suitable methods, protocols and system architectures which allow for interaction between multiple autonomous manipulation agents, such as a man-machine combination or a cognitive agent with AI capabilities and a lower-level automated manipulator such as a robot or autonomous vehicle. Clearly, there is also a need to bridge the gap between direct human control and AI control of manipulation systems, and a need for hand-off protocols between autonomous agents, whether human or machine. The problem and challenge presented by such activities is how to mediate the interaction of cognitive agents that are embedded in perception-cognition-action systems. This problem is exacerbated when communication delays are present in the control loop between the controlling agent and the controlling agent.
In light of the foregoing problems and needs, it is an object of the present invention to provide new methods and systems for improved man-machine interaction, including the use of improved machine manipulation visualization using forward simulations. It is another object of the present invention to provide methods and systems for real time control of a controlled agent such as a robot by a cognitive agent (human or AI) embedded in the robot's automation system. Yet another object of the present invention is to allow the operator in such a robotic system to generate interactively during an on-going real-time control situation a future path plan or segment which the robot will then follow.
Still another object of the present invention is to provide a means for and method for selectively engaging and disengaging time synchrony, position synchrony or both in a real time system employing forward simulation. One more object of the present invention is to improve further the efficiency of a man-machine system operating in real time to perform one or more tasks involving physical manipulation or movement. One first-related object is to provide means, for use in real time systems, which avoid jittery pre-positioning movements along a path in conjunction with fine or difficult operator-controlled manipulations. A second related object is to enable the robot to execute in real time and at near-optimum speeds path plans programmed only minutes or moments before by an embedded human agent in the real-time system.
Yet another object of the present invention is to provide a method and system operating in real time which allows a cognitive agent such as a human operator to intervene in the event that an obstacle presents itself in a path plan developed by a forward simulation. A related object is to provide means and methods for enabling tele-manipulation systems to handle unstructured tasks or tasks which due to events or circumstances that are not completely predictable cannot be managed entirely with conventional automated equipment operating in a simple pre-programmed manner.
Still another object of the present invention is to provide for a system and method for real time hand-offs and rendezvous of tasks between different cognitive or autonomous control agents, whether such agents be human or machine, operating in a common work environment.